Method and apparatus for operating a user interface

ABSTRACT

A method and apparatus for operating a user interface is provided, in which a first user input is detected, and a confidence analysis is carried out. Different kinds of system responses are generated depending on the result of the confidence analysis, wherein a first kind of system response comprises a system output which requires a second user input, and a second kind of system response comprises the automatic selection of a database entry. A third kind of system response comprises the first and second features, wherein the third kind of system response automatically changes into the first or second kind of system response after the expiry of a period of time. In this case, one of the two first kinds can be set as a standard or “default,” into which the automatic change then takes place after expiry of the period of time without further user interaction.

This nonprovisional application is a divisional of U.S. application Ser.No. 13/668,676, which was filed on Nov. 5, 2012, which was acontinuation of International Application No. PCT/EP2011/001776, whichwas filed on Apr. 9, 2011, and which claims priority to German PatentApplication No. DE 10 2010 019 191.4, which was filed in Germany on May4, 2010, and which are all herein incorporated by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a method for operating a userinterface, in which a first user input is detected, and a confidenceanalysis is carried out, during which the user input is compared with aplurality of stored database entries and each thus compared databaseentry is assigned a value of a confidence measure, which assesses thecorrespondence between the user input and this database entry. Differentkinds of system responses which differ from one another in at least onefeature are generated depending on the result of the confidenceanalysis, whereby a first kind of system response comprises, as adistinguishing first feature, a system output which requests a seconduser input, and a second kind of system response comprises, as adistinguishing second feature, the automatic selection of a databaseentry. The invention relates further to a corresponding apparatus,especially for carrying out this method, and to a vehicle having such anapparatus.

Description of the Background Art

Generic user interfaces have a broad range of applications for theoperation of devices, particularly electronic equipment. Particularly,the comparison of user inputs with database entries stored in a databaseare known from the technical field of search engines, in which a userinput is compared with database entries and the relevance of eachdatabase entry is calculated by means of an algorithm and an associatedconfidence measure is output, which quantifies the relevance. If such asearch result is unambiguous, the corresponding database entry is outputor a function is carried out on its basis. If the search result is notunambiguous, in this case either typically a hit list or a system-sidequery is output. In this regard, the user input and system output form adialog-like scheme, particularly when a system output responding to auser input asks for another user input.

Such user interfaces are used in particular when it is assumed thatdetected data are faulty either because of incomplete or ambiguousinputs on the part of the user or because of system-side imprecisedetection, as is the case, for example, in speech input in a noisyenvironment or in manual inputs in an environment affected byvibrations. For this reason, such interactive user interfaces are usedincreasingly in vehicles in order to improve the operating convenienceof the numerous operating devices, such as, for example, a navigationsystem, telecommunication devices, or infotainment systems.

Further, it must be taken into account in the case of a vehicle that thedriver is not distracted from the road traffic during an operatingprocedure via the user interface. It was proposed for this reason tomake possible a multimodal user input, whereby one can switch among thepossible input modalities to the one most advantageous for the user.Common input modalities include in particular inputs via speech,keyboard, touch screen, and non-contact gestures. DE 10 2008 008 948 A1describes, for example, such a method in which during a multi-part userinput a switch can be made between two input modalities during the inputas well.

DE 103 13 222 A1, which corresponds to U.S. Pat. No. 7,809,501,discloses a method for inputting place names in which ambiguous placenames are shown in a graphic, for example, in the correspondinggeographic positions on a map. The user then has the opportunity to makea rapid and unerring selection of the desired place name with the aid ofthis information.

DE 100 60 654 C1, which corresponds to U.S. Pat. No. 7,167,545,describes a method and a device for automatically issuing information bymeans of a search engine, in which the sought information, together withother associated attributes, are stored in the form of data records. Arelevance check is performed with the stored database entries for asearch argument input by the user for several attributes stored in adata record and via a weighting with confidence values the data recordwith the highest total confidence value is output. In the case ofseveral data records with a nearly equal confidence value, the user isasked for further user input on the data record until the sought datarecord was clearly identified.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide a methodand a corresponding apparatus for operating a user interface with systemresponses of a first and second kind with distinguishing first andsecond features of the aforementioned type, which have an improveddialog efficiency. In particular, the dialog duration and/or the numberof user interactions are to be reduced.

In an embodiment, the method can include a third kind of system responsehaving the first and second feature, i.e., the features of the first andsecond kind of system responses, whereby the third kind of systemresponse automatically changes into the first or second kind of systemresponse after the elapse of a period of time. System responses of athird kind with both features of the first and second kind have theadvantage that in the case of an ambiguous result of the confidenceanalysis both action options of the first and second type can beprovided to the user. In this case, one of the two kinds recognizable tothe user can be set as the standard or “default” to which the automaticchange then takes place after elapse of the period of time withoutfurther user interaction. The user can thus realize whether in the caseof an in fact ambiguous or incomplete input on his part the system hasrecognized the desired input. This is advantageous, when a further userinput, be it only a confirmation of the system-side suggestion,threatens to distract the user from another activity, for example,during driving of a vehicle.

The change in the system response can occur depending on the result ofthe confidence analysis. In particular, the result of the confidenceanalysis decides whether a change occurs from the third kind of systemresponse to the first kind of system response or to the second kind ofsystem response. Alternatively or also in addition, the result of theconfidence analysis for regulating the length of the time period or forconfiguring the change between the two kinds of system response can beincluded. For example, a change could contain a warning message that thesystem-side selection based on the result of the confidence analysis maybe incorrect. As a result, a rapid dialog flow can be weighed versus areduced risk of a faulty activation of a function.

In an embodiment of the method of the invention, depending on the resultof the confidence analysis, the activation of a function based on aselected database entry can occur immediately or after the elapse of atime period. The selection and activation of the function typicallyoccurs based on the database entry with the highest confidence measure.For example, in the case of a sufficiently unambiguous result of theconfidence analysis, the activation of a function occurs immediately,whereas in the case of a less unambiguous result thereby one waits forthe elapse of the time period. In the first case, this has the advantagethat the dialog flow is accelerated.

The time period after which the automatic change in the system responseoccurs without further user interaction is expediently dimensioned sothat an interaction by the user is possible. The duration of the timeperiod can be adjusted to the particular requirements and typically lieswithin the range of a few seconds to several, for example, 10 seconds.

Depending on the result of the confidence analysis, advantageously therequest for a second user input comprises a representation of databaseentries whose number is limited and/or whose assigned values of theconfidence measure are above a first threshold value. These so-calledhit lists show the user alternative suggestions, which he can selectquickly and conveniently from the indicated list, without having to makea complete new input.

In this case, the values of the confidence measure itself can also beshown or visualized. For example, the values of the confidence measureare presented as a percentage behind the associated database entry or,however, a database entry is highlighted in color depending on the valueor a size proportional to the particular value is selected. The user canvisually perceive and evaluate the result of the confidence analysisbetter in this way.

In an embodiment of the method of the invention, the database entriesare assigned positions on a map and the database entries are representedgraphically at the assigned position on the map. After the elapse of thetime period, then an area around the database entry with the highestconfidence measure is zoomed in automatically. This type of embodimentis especially advantageous for a navigation system in a vehicle.

In the case that the system-side selection of the database entry withthe highest confidence measure does not correspond to the desireddatabase entry that the user wanted to select by his user input, it isexpedient to integrate a correction option into the method of theinvention. Therefore, it can be provided according to the invention thatin the case of the third kind of system response before the elapse ofthe time period, a second user input can occur on whose basis theactivation of the function occurs. The second user input is inparticular a selection of a shown database entry whose confidencemeasure has one of the next highest values. It is immaterial in thiscase whether the function was already activated on the basis of apreviously wrongly selected database entry or this was provided for onlyafter the elapse of the time interval. In the first case, the functionexecution based on the wrongly selected database entry is expedientlyterminated.

The result of the confidence analysis is the key condition forcontrolling the method of the invention. It is assumed in a confidenceanalysis that data detected on the system side, e.g., by incomplete orambiguous entries or by imprecise detection are faulty, as is the case,for example, in speech input in a noisy environment or in manual inputsin an environment affected by vibrations. In particular, the result ofthe confidence analysis in this case can be unambiguous or lessunambiguous. Within the meaning of the invention, there is an idealunambiguous result of the confidence analysis, when precisely onedatabase entry with a very high confidence measure is determined. A lessunambiguous result is characterized by the fact that, for example, no ora plurality of database entries with a very high confidence measure aredetermined.

The confidence measure in this case can be calculated in a different wayand, for example, expressed as a percentage, as is known, e.g., from thefield of search engines. The calculation basis used in the specific caseis not critical and can be adapted to the circumstances present. If thedatabase entries concern individual words, a comparison could be made ona letter basis and the correspondence of the letters enter into theconfidence measure as a percentage. Further, the confidence measure forfrequently selected database entries could be increased, when anincreased probability of a new selection can be assumed here.

The result of the confidence analysis according to the inventiontherefore typically comprises whether the highest value of theconfidence measure exceeds a second threshold value. This secondthreshold value typically is very high close to a possible optimum(e.g., >90%). Alternatively or also in addition, the result alsoincludes how many values of the confidence measure exceed this secondthreshold value. Alternatively or also in addition, it can be taken intoaccount whether the difference between the highest and one of the nexthighest values of the confidence measure falls below a third thresholdvalue. This third threshold value is typically small, particularly smallcompared with the second threshold value (e.g., <10%).

It can be provided in embodiments of the method of the invention thatthe type of graphic presentation depends on the result of the confidenceanalysis. For example, a hit list depending on the result can be shownthe smaller the more unambiguous the result.

Alternatively or also in addition, depending on the results of theconfidence analysis, a system response can comprise further features.For example, acoustic or haptic signals can be output to make the useraware multimodally of the specific result.

If a plurality of display surfaces or a very large display surface isavailable, depending on the result of the confidence analysis, thesystem output can occur on the different display surfaces or atdifferent positions on a display surface. If based on the confidenceanalysis with high probability a correctly selected database entry is tobe assumed, a hit list can be shown at a less prominent position on thedisplay surface or on a secondary display surface, which is advantageousparticularly when a function was already activated in parallel, whichrequires a major part of the display surface itself. As a result, therisk of distraction for the user is reduced, particularly when it isassumed that the user would not like to see another selection or wouldnot like to enter a correction.

The apparatus of the invention for operating a user interface comprisesmeans for detecting user inputs, an interface to a functional device inwhich database entries are stored, and a display surface for the visualpresentation of system outputs. It comprises further a control unit forcarrying out a confidence analysis, whereby comparisons can be madebetween user inputs and a plurality of stored database entries and eachof the compared database entries can be assigned a value of a confidencemeasure, which assesses the correspondence of the user input to thisdatabase entry. Different system responses which differ from one anotherin at least one feature can be generated by the control unit dependingon the result of the confidence analysis, whereby a first kind of systemresponse comprises, as a distinguishing first feature, a system outputwhich requests a second user input, and a second kind of system responsecomprises, as a distinguishing second feature, the automatic selectionof a database entry. In this case, the control unit can generate a thirdkind of system response which comprises the first and second feature,whereby the third kind of system response automatically changes into thefirst or second kind of system response after the elapse of a period oftime. The apparatus is particularly suitable for carrying out the methodof the invention. Thus, it also has the advantages of the method of theinvention.

According to an embodiment of the invention, a vehicle is equipped withsuch an apparatus for operating a user interface.

Further scope of applicability of the present invention will becomeapparent from the detailed description given hereinafter. However, itshould be understood that the detailed description and specificexamples, while indicating preferred embodiments of the invention, aregiven by way of illustration only, since various changes andmodifications within the spirit and scope of the invention will becomeapparent to those skilled in the art from this detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more fully understood from thedetailed description given hereinbelow and the accompanying drawingswhich are given by way of illustration only, and thus, are not limitiveof the present invention, and wherein:

FIG. 1 shows schematically the structure of an exemplary embodiment ofthe apparatus of the invention for operating a user interface;

FIGS. 2a-2c show graphic illustrations of database entries, which weregenerated according to an exemplary embodiment of the method of theinvention;

FIG. 3 shows schematically the structure of a hit list, which wasgenerated according to an exemplary embodiment of the method of theinvention; and

FIGS. 4a-4b show schematically the assignment of the kinds of systemresponses depending on the result of the confidence analysis accordingto an exemplary embodiment of the method of the invention.

DETAILED DESCRIPTION

The exemplary embodiment described below relates to the use of theapparatus and of the method in a vehicle, particularly a motor vehicle.A user interface in the vehicle can be provided to the vehiclepassengers by way of the method and the apparatus. It is pointed out,however, that the apparatus and the method can be used in the same wayin other devices as well, such as, for example, portable devices.

The structure of an exemplary embodiment of the apparatus of theinvention for operating a user interface is illustrated schematically inFIG. 1. A display surface 10 in a vehicle is connected via a controlunit 13 to means 11 for the manual input and means 12 for speech input.Control unit 13 is connected via an interface 14 to data bus 18 in thevehicle, by way of which the control unit has access to a database 15.Database 15 in this case can be a comprehensive database in which alldata accumulating in the vehicle for the diverse functional devices canbe retrieved. Each functional device comprises in addition a separatedatabase 15 in which the data specific to the functional device arestored. The functional devices are, for example, a permanently installednavigation system 16 or also one that can be removed from the vehicleand a functional system 17 with a wireless interface, e.g., a cellulartelephone.

The means 12 for speech input comprise a microphone for receivingacoustic signals, a software module for voice activity detection, and anadaptive filter for noise cancellation. The voice activity detection andthe adaptive filter alternatively can also be localized in otherhardware and be interconnected, for example, via control unit 13 to themeans 12 for speech input. The means 11 for the manual input can beconfigured as a keyboard or manual operating elements, e.g., rotatingpush button. Display surface 10 may have totally or in areas atouch-sensitive surface, so that manual inputs can be made also via thedisplay surface. For example, a keyboard can be displayed on a sectionof display surface 10 and manual input can occur by touching of adisplayed key.

Different information can be displayed graphically on display surface10. In particular, the information concerns database entries 21 whichare stored in one of the databases 15 and are displayed according to themethod of the invention, described further in detail below, on ageographic map or also in the form of a hit list 20. In the shownexemplary embodiment, display surface 10 is divided into a primarydisplay surface 10 a, which is configured as a large display, and into asecondary display surface 10 b. Display surfaces 10 a and 10 b in thiscase can be separated physically and spatially from one another. Inparticular, the primary display surface 10 a extends over wide parts ofthe cockpit in the vehicle, so that, for example, it extendssubstantially over the entire width of the vehicle or is projected ontothe windshield as a so-called head-up display.

Control unit 13 is configured to carry out the method of the invention.It comprises a processor for running a program to perform a confidenceanalysis and for controlling system responses and a data memory forstoring the results.

The method of the invention will now be explained in greater detail withthe use of exemplary embodiments with reference to FIGS. 2-4. Whereas inthis case illustrations of the system responses are described withreference to FIGS. 2a to 2c and 3, the confidence analysis will bediscussed in greater detail with reference to FIGS. 4a and 4b . Theresult of the confidence analysis can be assigned basically in the mostdiverse way to different representations of the system responses,without being limited to the assignments described below.

Graphic representations with visualizations of database entries, whichwere generated according to an exemplary embodiment of the method of theinvention, are shown in FIGS. 2a to 2c . The graphic contents of anavigation system 17 are output on display surface 10 configured as atouch screen in a vehicle. The user, for example, the driver of thevehicle, would like to input “Berlin” as a new destination address. Tothis end, he is shown a keyboard on display surface 10, which he canoperate conveniently by simple touching display surface 10. If he mustinterrupt his user input, e.g., because of some other necessary processfor driving the vehicle, the system performs a comparison on the basisof this possibly incomplete entry with all database entries 21 in thenavigation system and assesses each possible hit with a confidencemeasure 22.

Alternatively, the user input could also have occurred using speech.Here, the results could have been especially ambiguous, because a highbackground noise level prevails in the vehicle or the user does notspeak clearly. The database comparison could be made in a similar way.In particular, the user input can have several parts in that, forexample, street and place names are input or the individual parts of theuser input occur multimodally, e.g., manually and via speech, as isdescribed in the previously cited DE 10 2008 008 948 A1.

In the confidence analysis, in particular the input letters “Ber,” forexample, are compared with available database entries 21. In addition,for example, it can be considered which destinations were recentlydriven to, what the current position of the vehicle is, or whether otherattributes of the user input are known, for example, when a street namewas specified in a previous search or in a multi-part input.

In the exemplary embodiment, several possible hits are found whosedetermined values of the confidence measure 22 are high. Positions 28 onthe map are assigned to the corresponding database entries 21 andgraphically represented at positions 28. Database entry 21 a with thehighest confidence measure 22 is selected. On the system side, nowfurther user input is awaited either in form of a confirmation ofselected database entry 21 a or in the form of a correction input.

If the user realizes that the selected database entry 21 a is not thedesired destination, he can input a suitable correction, in particularselect one of the other displayed database entries 21 by touching theparticular position 28 on the touch screen. If after the elapse of atime period, no further user input occurs, an area around position 28 aof the already selected database entry 21 a is zoomed in automatically,as shown in FIG. 2b . In this regard, the time period can be selectedthe shorter, the more unambiguous the result of the confidence analysis,particularly when as in the shown example only three possible resultswere identified.

It is shown in FIG. 2c how after the zooming in on an area around thedesired city the multi-part user input is compared once more withdatabase entries 21 and now to identify the desired street again threepossible hits are shown on the map at the assigned positions 28. Here aswell, a database entry 21 a is again selected and after the elapse of atime period without user input an area around position 28 a around theselected database entry 21 a would be zoomed in automatically (notshown).

The schematic structure of a hit list 20, which was generated accordingto another exemplary embodiment of the method of the invention, is shownin FIG. 3. Hit list 20 comprises two columns, in which relevant databaseentries 21 are listed in the left column and the associated calculatedvalues of confidence measure 22 are compared in the right column.Database entries 21 are sorted in decreasing order of the assignedconfidence measure 22. The confidence measure in the exemplaryembodiment has values between 0≦C≦1. A first threshold value is set toC=0.8, whereby in the displaying of a hit list only database entrieswith a confidence measure above this first threshold value areindicated. In addition, the display of database entries can be limitedto a fixed number, for example, to 10 hits, to assure a clear displayand not to overtax the user, particularly the driver of the vehicle,visually.

The assignment of the kinds of system responses depending on the resultof the confidence analysis according to an exemplary embodiment of themethod of the invention is shown schematically in FIGS. 4a and 4b . Theresult of the confidence analysis is categorized in a results table 24as a 3×3 matrix, in that two criteria are tested for exceeding orfalling below two threshold values. The first criterion is the testingwhether the highest occurring value C of confidence measure 22 exceedsthe second threshold values C=0.8 and C=0.9. The second criterion is thetesting whether the value ΔC of the difference 23 between the highestand second highest value C of the confidence measure 22 falls below thethird threshold value ΔC=0.1 and ΔC=0.2.

The result of the confidence analysis is the more unambiguous, thefurther right and further to the top the result is in results table 24.Thus, in matrix field A13 the confidence measure of the most relevantdatabase entry is high (>0.9) and the distance to the next databaseentry is also high (>0.2). Accordingly, the result in matrix fieldsA31-A33 is ambiguous.

It is therefore sensible to assign the ambiguous search resultsrepresented by matrix fields A31-A33 to a system response of the firstkind I, defined by the value range 25, which asks for a second userinput. It cannot be assumed in this case based on the first user inputthat the desired database entry can be accurately recognized andselected system-side. To avoid false activations, a second user input inthe dialog flow can therefore lead to the desired result more rapidly.In particular, the second user input can be supported and accelerated inthat database entries 21 with the highest values of confidence measure22, as described, for example, in FIG. 3, are shown, which increases therecognition and the convenience during the second user input, e.g., aselection by touching the touch-sensitive display surface 10.

Similarly, an unambiguous search result, represented by the matrix fieldA13, can be assigned to a system response of the second kind II, definedby the value range 26. Based on the first user input it can be assumedthat the desired database entry can be recognized accurately andselected system-side. To avoid unnecessary further system-side queriesto the user, therefore immediately based on the selected database entry21 a function can be carried out, which accelerates the dialog flow andrapidly leads to the desired result.

The other matrix fields are assigned to a system response of the thirdkind III, defined by the value range 27. In this case, both a selectionof database entry 21 with the highest confidence measure 22 occurs andalso a system-side request for a second user input. In this case, as inthe shown example, the system response of the third kind III can bedivided into several value ranges 27 a and 27 b. The result for theconfidence analysis is more unambiguous for value range 27 a than forvalue range 27 b. It is provided therefore for value range 27 a togenerate a system response of the kind IIIa, in which based on theselected database entry 21 a a function is carried out immediately,whereas for value range 27 b a system response of the kind IIIb isgenerated, in which the function is carried out only after the elapse ofa time period. In both cases, a hit list 20, for example, according toFIG. 3, can be displayed as support to accelerate a possible second userinput as a correction for a wrongly selected database entry 21 a. Afterthe elapse of a time period, hit list 20 is again hidden.

The method of the invention can be designed in a very different way tobe adapted to the particular circumstances. The adaptations relateparticularly to the input and output modalities, the assignment to thekind of system responses depending on the confidence analysis, and thespecific design of the presentation. They can be preset in this case andoptimized for the application or also be configurable by the userhimself.

The input and output modalities for the user input and the system outputcan be adapted ergonomically to the particular application. Inparticular for operation in a vehicle, user input via speech is suitablein the method of the invention, whereby, however, particularlymultimodal inputs with inclusion of gestures and via manual operatingelements are advantageous. Voice interfaces are less suitable inso-called quiet zones in which the noise level is to be kept low, or forusers with hearing or speech disorders. In this case, the method of theinvention can be used furthermore in that manual user inputs areprovided. Likewise, a system output can be adapted multimodally to thecircumstances.

The classifying of the system responses can occur in many ways. Thus,more than two criteria can be considered for the result of theconfidence analysis and thereby many threshold values per criterion canbe defined. In this way the system can respond in an even moredifferentiated manner to a user input and thereby optimize theperformance and reaction time. Particularly the time period after whicha change in a system response of the third kind becomes a systemresponse of the first or second kind can be calculated from the highestvalue or the highest values of the confidence measure or sums anddifferences thereof. These values can be adjusted particularly to theapplication program for which the user interface is provided.

A function execution can be activated more rapidly for applicationprograms with very long response times than in application programs withvery short response times. If, for example, the method of the inventionis used to search for a contact person who is stored in a cellulartelephone and to whom a call over the cell phone is to be made, this canperhaps require some time in a vehicle at a rather high speed, becausethe cell phone must first reserve transmission resources in the localwireless cell and second the other party, optionally also a cell phone,must first be localized by the network and the connection must be madethere. In this case, the connection could be set up in advance, alsowhen the result of the confidence analysis was not very unambiguous. Ifthe wrong party was selected, the attempted call can always still beterminated in time by the correction option and another telephone numberselected.

The invention being thus described, it will be obvious that the same maybe varied in many ways. Such variations are not to be regarded as adeparture from the spirit and scope of the invention, and all suchmodifications as would be obvious to one skilled in the art are to beincluded within the scope of the following claims.

What is claimed is:
 1. A method for operating a user interface,comprising: detecting a first user input; performing a confidenceanalysis, during which the first user input is compared with a pluralityof stored database entries and each so compared database entry isassigned a value of a confidence measure, which assesses acorrespondence between the first user input and the database entry; andgenerating different kinds of system responses, which differ from oneanother in at least one feature, depending on a result of the confidenceanalysis, the system responses comprising: a first kind of systemresponse comprising, as a distinguishing first feature, a system outputwhich requests a second user input; a second kind of system responsecomprising, as a distinguishing second feature, an automatic selectionof the database entry; and a third kind of system response comprisingthe first and second feature, wherein the third kind of system responseautomatically changes into the first kind or system response or thesecond kind of system response after a period of time has elapsed, andwherein with a sufficiently unambiguous result of the confidenceanalysis, a function on the basis of a selected database entry isimmediately activated, whereas with a less unambiguous result of theconfidence analysis, the function on the basis of a selected databaseentry is activated after a further period of time has elapsed.